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SUMMARY 

The  effectiveness  of  a command  centre  largely  depends 
on  the  effectiveness  of  the  team  that  keeps  it  going.  This 
paper  describes  a framework  of  five  research  methods  to 
investigate  command  centre  teams.  These  methods 
comprise  modelling,  observing,  experimentation,  design, 
and  evaluation.  Modelling  implies  breaking  up  a whole 
human-human-machine  system  into  essential  elements.  It 
is  an  analysis  resulting  in  a clear  description  of  the 
system.  Observation  is  needed  to  identify  possible 
bottlenecks  in  the  command  centre.  It  yields  insight  in 
the  composite  set  of  factors  that  influence  the 
effectiveness  of  the  command  centre  team.  Single  factors 
can  be  investigated  systematically  by  experimentation 
using  a contrived  experimental  task.  The  knowledge  that 
is  gained  by  modelling,  observation  and  experimentation 
can  lead  to  the  design  of  a new  command  centre  team,  or 
the  redesign  of  current  ones.  Finally,  any  particular 
design  may  need  an  evaluation  to  determine  how  team 
performance  is  effected.  The  application  of  the 
framework  is  illustrated  by  a number  of  research 
projects. 
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1 INTRODUCTION 

The  ability  of  teams  to  work  effectively  is  a prerequisite 
in  a number  of  critical  work  environments,  including 
military  command  centres,  fire-fighting,  aircraft 
cockpits,  emergency  medicine,  and  air  traffic  control. 
Such  teams  often  have  to  perform  in  complex  situations 
that  are  characterised  by  time  pressure,  heavy  workload, 
ambiguous  information  presentation  and  a constantly 
changing  environment.  High  stakes  are  involved  and 
poor  performance  may  lead  to  considerable 

consequences. 

Many  studies  demonstrate  the  importance  of  teamwork. 
For  example,  in  the  aviation  domain  several  studies  have 
shown  that  many  incidents  and  accidents  are  due  to 
miscommunication  of  the  flight  crew  (Helmreich  & 
Foushee,  1993).  The  accident  with  the  USS  Vincennes 
has  been  attributed  to  ineffective  teamwork  (Klein, 
1993).  Heath  & Luff  (1992)  show  that  effective  crisis 


management  in  the  London  underground  line  control 
room  depends  on  how  operators  exchange  and  monitor 
information.  Finally,  in  the  medical  world,  ineffective 
teamwork  has  led  to  a considerable  number  of  incidents 
in  anaesthesia  (Howard,  Gaba,  Fish,  Yang,  & Samquist, 
1992)  These  studies  show  the  importance  of 

investigating  teamwork  to  identify  the  factors  that  makes 
a team  effective. 

The  focus  of  this  paper  is  the  research  on  command 
centre  teams.  A command  centre  team  is  defined  as  a set 
of  at  least  two  people  that  work  together  toward  a 
common  goal,  who  have  been  assigned  to  specific  roles 
or  tasks  to  be  performed,  and  where  the  completion  of 
the  mission  requires  dependency  among  team  members 
(Dyer,  1984;  Salas,  Dickinson,  Converse  & 

Tannenbaum,  1992).  Central  in  our  approach  is  that  the 
command  centre  is  viewed  as  a complex  cognitive 
system  in  the  sense  that  it  takes  situation  specific 
information,  knowledge  from  training  and  experience, 
mental  constructs  (hypotheses  and  assumptions),  and 
norms  and  values  that  are  combined  into  new 
information  entities  (Essens,  Post,  Rasker,  2000).  The 
core-business  of  a team  in  the  command  centre  is 
information.  Signals,  datalinks,  intelligence  and  so  forth 
must  be  received,  interpreted,  and  assessed  in  order  to 
decide  upon  the  righteous  action.  Because  tasks, 
information  sources,  and  expertise  is  assigned  to 
different  team  members,  interdependent  interaction  is 
needed.  This  interdependency  requires  team  members  to 
engage  in  teamwork  behaviours  such  as  communication 
and  co-ordination.  The  challenge  is  to  determine  how- 
effective  teamwork  can  be  realised  in  complex  systems 
such  as  the  command  centre. 

The  appreciation  that  teamwork  must  be  seriously 
studied  is  shown  by  the  variety  of  research  questions  that 
come  from  practice.  A first  type  of  question  is  related  to 
the  complexity  of  a command  centre  representing  a need 
for  a clear  description  and  understanding  of  this  human- 
human-machine  system.  Questions  such  as,  “which  tasks 
are  performed  and  which  knowledge  is  needed”  and 
“what  is  the  information  flow  in  the  command  centre  and 
how  is  this  exchanged  by  team  members”  represent  this 
need.  A second  type  of  question  arises  out  of  problems 
that  are  experienced  in  practice.  Often  there  is  a vague 
idea  that  something  is  wrong,  but  one  cannot  lay  hold  on 
the  specific  bottlenecks  and  the  seriousness  of 
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bottlenecks.  This  is  represented  by  questions  such  as 
“are  there  problems  with  the  workload?”  or  ”do  we  train 
our  personnel  well  enough?”.  A third  type  of  question 
comes  from  a need  to  develop  a deeper  understanding  in 
the  underlying  factors.  Questions  such  as  “which  type  of 
knowledge  is  needed  to  communicate  efficiently”  or 
“what  is  the  effect  of  cross-training  on  co-ordination” 
represent  this  need.  A forth  type  of  question  is  associated 
with  the  design  of  new  concepts,  systems  or  layouts. 
This  is  represented  by  questions  such  as  “how  must  the 
command  centre  be  equipped”  and  “what  is  the  best 
layout  of  the  command  centre  to  support  effective 
human-human  interaction”.  Finally,  the  last  type  of 
question  arises  from  a need  to  evaluate  whether  the 
designed  concepts,  systems  and  layouts  are  applicable 
and  effective  in  the  real  world. 

The  objective  of  this  paper  is  to  describe  in  a framework 
how  several  research  methods  can  be  applied  to 
investigate  command  centre  teams.  These  methods  are 
used  to  find  answers  for  the  various  types  of  questions 
stated  above.  With  these  methods,  teamwork  can  be 
investigated  in  command  centres  as  well  as  in  any  other 
complex  human- human-machine  system.  We  will 
illustrate  the  framework  with  a number  of  research 
projects  and  research  techniques.  First  the  framework  as 
a whole  is  explained,  and  then  each  method  is  discussed 
separately. 

2 RESEARCH  FRAMEWORK 

For  investigating  team  effectiveness  in  complex  systems 
we  describe  a research  framework  consisting  of  five 
complementary,  but  individually  applicable,  methods 
that  span  all  phases  of  team  research  and  development. 
The  five  methods  are  modelling,  observation, 
experimentation,  design,  and  evaluation.  Figure  1 
presents  this  research  framework. 


Modelling  is  needed  for  gaining  understanding  of  the 
complex  environment  in  which  the  team  has  to  operate. 
Modelling  yields  a description  of  the  essentials  of  a 
team,  such  as  its  structure , the  links  with  the  broader 
organisation  in  which  the  team  has  to  operate,  the 
functions  that  has  to  be  fulfilled,  the  tasks  that  the  team 
members  have  to  carry  out,  the  knowledge  they  need  for 
that,  and  the  types  of  information  that  flows  within  the 
team.  The  resulting  team  model  provides  the  set  of 
descriptors  for  all  team  entities  that  are  subject  of  further 
team  research  and  development.  Once  the  team  is  well 
enough  understood,  one  is  able  to  make  a useful 
observation  of  an  operating  team.  Observation  deals  with 
the  actual  team  process,  revealing  a problem  description 
in  terms  of  possible  bottlenecks  and  challenges  for 
improvements.  During  observations,  many  facets  play  a 
role  at  the  same  time,  and  linking  a problem  to 
(interaction  of)  particular  jactor  is  not  always  possible. 
With  experimentation,  where  a team  environment  can  be 
controlled,  the  contribution  of  separate  factors  for  team 
effectiveness  can  be  identified.  New  initiatives,  or 
observed  bottlenecks  and  challenges  in  an  existing  team 
lead  to  team  (re)design.  Design  encompasses  the 
application  of  all  factors  of  team  effectiveness  (known 
from  own  research  as  well  as  others)  to  new  concepts 
and  detailed  realisations.  Finally,  a designed  team  may 
be  evaluated  to  acquire  objective  team  performance 
measures  in  terms  of  speed,  workload,  quality,  and 
quantity. 

Each  method  has  a set  of  research  and  development 
techniques  associated  to  it.  The  next  sections  describe  de 
application  of  each  method  in  several  projects,  and  how 
they  are  supported  by  the  available  techniques. 

3 MODELLING 

The  functioning  of  a command  centre  team  is  not  easy  to 
understand.  For  an  outsider  it  probably  seems  like  magic 
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Figure  1 : Research  framework  for  team  research  and  development 
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how  a team  does  its  job,  using  an  overwhelming  amount 
of  displays,  push  buttons  and  communication  equipment, 
communicate  with  strange  abbreviations,  being  quiet  at 
one  time,  and  completely  occupied  the  next  moment. 
The  purpose  of  modelling  is  to  develop  an  understanding 
of  this  complex  team  and  its  environment.  This  is  done 
by  reducing  its  complexity  through  an  analysis  and 
description  process.  By  analysis,  the  whole  system  is 
broken  up  into  its  essentials.  This  is  a complex  process 
itself,  because  the  essentials  and  their  relationships  are 
not  readily  given.  Often,  they  have  to  be  abstracted  from 
the  many  details,  and  the  abstract  entities  that  arise  need 
to  be  labelled  and  described.  By  doing  so,  modelling 
provides  a clear  team  description  and  an  easily 
accessible  “map”  of  the  command  centre  team. 

There  are  many  ways  to  model  a team.  In  our  approach 
we  distinguish  nine  so-called  team  modelling 
perspectives  (Essens  et  al.,  2000).  These  are  an 
organisational  model,  a function  model,  an  information 
model,  a function-information  model,  a human  agent 
model,  a technical  means  model,  a knowledge  model,  a 
task  model,  and  an  event  handling  model  (See  figure  2). 
The  organisation  model  shows  the  relation  of  the  team 
within  the  broader  organisation.  The  function  model 
shows  the  command  centre  functions  hierarchically.  The 
information  model  shows  the  information  that  is  used 
and  processed  in  the  command  centre  hierarchically.  The 
function-information  model  shows  the  information 
dependencies  of  the  functions  on  the  different 
hierarchical  levels.  The  agent  model  describes  the  team 
member  roles;  the  technical  means  model  the  equipment 
that  team  members  have  to  their  disposal.  The  task 
model  takes  the  end  nodes  of  the  functional  hierarchy 
and  couples  these  with  the  end  nodes  of  the  information 
hierarchy  (as  input  and  output),  the  technical  means, 
agents,  and  the  knowledge  entities  as  controls.  Finally, 
the  event-handling  model  shows  the  response  to  a 
tactical  event  in  a temporal  sequence  of  functions  that 
are  distributed  over  agents. 


By  using  a hierarchical  approach  in  modelling,  we  are 
able  to  finish  at  any  hierarchical  level  and  still  have  a 
complete  description.  If  necessary,  specific  end  nodes  of 
the  hierarchy  can  be  modelled  in  more  detail  when 
bottlenecks  are  experienced  (or  established  by 
observation)  in  that  part  of  the  hierarchy. 

Project 

This  method  is  applied  to  model  the  command  centre  of 
the  Royal  Netherlands  Navy  (RNLN)  multipurpose 
frigate.  Knowledge  about  the  task-related  structure  was 
elicited  from  domain  experts  during  group  sessions.  This 
forced  the  researchers  to  specify  precisely  their 
understanding  of  the  system.  The  large  quantity  of 
information  (changes,  annotations,  new  branches) 
elicited  in  the  group  sessions  was  difficult  to  process 
afterwards.  Because  the  models  are  strongly  interrelated, 
adapting  them  became  a major  effort.  For  future 
modelling  efforts,  further  technical  support  is  needed  for 
consistency  checking,  glossary  update,  version 
management,  and  for  having  navigational  overview 
within  and  between  models.  Overall,  this  provided  us 
with  approximately  400  interrelated  graphical 
representations  implemented  with  a browser  on  CD- 
ROM. 

Techniques 

The  first  technique  applied  in  modelling  is  document 
analysis.  We  gathered  various  documents,  built  a 
glossary  of  the  terminology  found,  and,  after  having 
studied  the  material,  we  started  with  representing  our 
own  mental  model  of  the  human-human  system.  Next, 
we  developed  a visualisation  technique  to  describe  this 
mental  model,  first  for  communication  among  the 
researchers.  We  used  a representation  language  and 
graphics  in  the  models  consisting  of  a restricted  set  of 
descriptors  with  a consistent  form  and  a consistent 
meaning.  For  example,  in  figure  3,  an  arrow  means  data 
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dependency,  a small-circle  with  a line  means  a part-of 
relationship,  a rounded  box  represents  a function  or  task, 
a square  box  represents  an  information  entity,  and 
knowledge  is  represented  by  a square  box  with  a line, 
After  this  preliminary  description,  we  used  the  graphical 
model  as  a means  for  interviewing  the  domain  experts. 
Because  many  subjective  perspectives  arose  with  the 
various  experts  we  interviewed,  we  finally  asked  other, 
senior,  experts,  to  give  the  models  an  official  status,  a 
technique  we  called  standardisation. 


4 OBSERVATION 

Observations  are  needed  to  give  an  answer  to  the 
questions  that  arise  out  of  problems  that  are  experienced 
in  practice,  but  were  one  cannot  identify  the  specific 
bottlenecks  or  the  seriousness  of  these  bottlenecks.  It 
concerns  an  issue  description  in  order  to  develop  insight 
in  the  challenges  to  improve  team  effectiveness.  With 
observation  we  mean  a systematic  analysis  and  on-line 
measurements  of  the  ongoing  process  of  team 


Figure  3:  example  of  one  level  of  the  function-information  model 


Figure  3 shows  an  example  of  the  used  representation 
language  for  the  function-information  model.  This 
example  describes  Command  & Control  at  a generic 
level  (Passenier  & van  Delft,  1997).  The  functions  are 
linked  with  their  information  input  and  output  entities. 
The  diagram  shows  the  data  dependencies  between  the 
functions.  Situation  awareness  consists  of  a permanent 
process  of  compilation,  monitoring,  and  maintenance  of 
the  actual  situation.  Outside  information  and  internally 
perceived  information  (i.e.,  from  own  sensors)  is  used  to 
develop  and  maintain  an  actual  situation  description. 
This  information  is  used  for  threat  assessment  by 
interpreting  the  current  situation  from  a tactical 
perspective.  Planning  <£  decision-making  is  performed 
by  selecting  options  and  planning  (counter)measures. 
The  decision  is  executed  during  direction  & control  by 
giving  directives  and  allocating  resources. 


functioning  in  its  natural  setting.  This  provide  insights  in 
the  effectiveness  of  the  command  centre  team  and  reveal 
possible  bottlenecks  concerning  individual  task 
performance,  communications  and  information 
exchange,  co-ordination,  workload  and  so  forth.  This 
insight  is  needed  to  solve  the  experienced  problems 
either  by  designing  a new  system  or  by  adjusting  the 
current  system. 

Project 

Observations  are  performed  to  investigate  the  command 
centre  of  the  RNLN  multipurpose  frigate  (see,  for  an 
elaborate  description,  Essens,  this  volume).  Four 
operational  command  centre  teams  performed  three 
realistic  warfare  scenarios  (e.g.,  baseline  normal 
pressure,  time  pressure  from  handling  multiple  objects, 
uncertainty  form  object  behaviours)  in  a simulated 
command  centre  trainer  with  a duration  of  three  hours 
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including  operational  briefing  at  start  shift.  Seven 
domain  experts  strategically  distributed  over  the  team 
members’  positions  determined  systematically  the 
performance  quality.  During  task  execution  each  team 
member  produced  a workload  score.  A questionnaire 
completed  data  collection.  In  a debrief  session  a 
selection  of  the  observations  were  discussed  with  the 
experts,  the  investigators,  and  team  members  to  get  more 
insight  in  the  background  of  particular  actions  and 
decisions.  Based  on  these  observations  it  was  concluded 
that  in  times  of  stress  own  task  responsibilities  dominate 
and  push  away  teamwork  responsibilities  despite  the 
generally  strong  emphasis  on  total  team  performance. 

Techniques 

To  observe  systematically,  three  techniques  are 
developed  to  measure  performance:  a quality  indicator 
sheet,  a workload  registration  tool,  and  a task- 
performance  questionnaire.  In  addition,  task 

performance  is  registered  digitally  on  video  (including 
audio)  that  is  used  for  reviewing. 

After  a pre-briefing  about  the  content  of  the  research, 
performance  is  measured  during  task  execution.  Domain 
experts  (e.g.,  operational  experienced  officers)  score  the 
quality  indicator  sheet.  Each  time  a problem  is  scored, 
the  experts  also  push  a button  that  results  in  a time 
registration  linked  to  the  video  stream.  This  is  used 
during  the  review  to  the  find  easily  the  video  data  that  is 
linked  to  the  scored  problem.  Workload  is  measured 
with  the  workload  registration  tool.  This  specially 
designed  button  panel  consists  of  five  buttons.  The 
buttons  correspondent  with  a five  point  scale  that  each 
measures  a degree  of  subjective  workload.  The 
registration  of  the  subjective  workload  takes  place  every 
five  minutes.  The  button  panels  are  linked  to  the  video 
stream  that  registers  the  measurements. 

Afterwards,  the  team  members  fill  in  the  task- 
performance  questionnaire.  In  the  meantime,  the  domain 
experts  and  the  researchers  review  and  discuss  the  task 
performance  based  on  the  scores  on  the  quality  indicator 
sheet.  During  this  review  it  is  determined  which 
observed  problems  are  the  most  critical  and  need  to  be 
made  more  explicit.  Subsequently,  the  domain  experts, 
the  researchers,  and  the  team  members  discuss  these 
problems  to  develop  a deeper  understanding  of  the 
underlying  causes.  When  needed,  the  video  data  are 
showed  to  explicate.  All  together  the  experts’  scores,  the 
workload  measures,  the  questionnaires,  and  the 
discussion  reports  form  the  data  from  which  conclusions 
about  the  observed  problems  are  drawn. 

5 EXPERIMENTATION 

Observation  yields  insight  in  the  composite  set  of  factors 
that  influence  the  effectiveness  of  the  command  centre 
team.  From  observations,  however,  it  is  not  clear  to  what 
extent  single  factors  may  affect  team  effectiveness. 


Therefore,  we  investigate  single  factors  systematically 
by  using  a contrived  experimental  task  in  the  form  of  a 
low  fidelity  simulator.  With  the  help  of  such  a task  we 
attempt  to  identify  factors,  and  develop  an  understanding 
of  their  effects  and  interactions.  There  are  several 
advantages  for  using  a low-fidelity  simulation  (see  also, 
Bowers,  Salas,  Prince  & Branmck,  1992).  First,  the 
methodology  is  available  at  relatively  low  cost.  Second, 
it  gives  opportunities  to  develop  the  characteristics 
needed  in  team  research.  Third,  it  provides  the  requisite 
experimental  control  of  independent  variables.  Finally,  it 
is  relatively  simple  to  train  team  members,  which  makes 
it  possible  to  invite  naive  participants  in  stead  of 
operational  team  members  that  are  difficult  to  recruit. 

Project 

Experimentation  is  used  to  investigate  the  effect  of  intra 
team  feedback  on  developing  shared  mental  models  in 
Command  & Control  teams  (Rasker,  Post  & Schraagen, 
in  press).  Team  members  that  share  mental  models,  are 
expected  to  have  common  expectations  of  the  task  and 
the  team,  allowing  them  to  predict  the  information  needs 
of  team  members  accurately.  This  makes  it  possible  for 
team  members  to  offer  each  other  the  necessary 
information  at  the  right  moment,  which  results  in  an 
improved  performance.  It  is  hypothesised  that  intra  team 
feedback  plays  an  important  role  in  adjusting  and 
developing  a shared  mental  model.  By  giving  each  other 
feedback,  team  members  develop  an  understanding  of 
each  other’s  tasks  that  gives  them  insights  in  which  and 
when  information  must  be  exchanged.  A distinction  is 
made  between  performance  monitoring  and  team  self- 
correction.  Performance  monitoring  is  the  ability  of  team 
members  to  monitor  each  other’s  task  execution  and  give 
feedback  during  task  execution.  Team  self-correction  is 
the  process  in  which  team  members  engage  in  evaluating 
their  performance  and  determining  their  strategies  after 
task  execution.  In  two  experiments,  the  opportunity  to 
engage  in  performance  monitoring  respectively  team 
self-correction  was  varied  systematically.  The  results 
show  that  teams  perform  better  when  they  have  the 
opportunity  to  engage  in  intra  team  feedback.  Both 
performance  monitoring  as  well  as  team  self-correction 
appeared  to  be  beneficial  for  team  performance. 

Techniques 

One  complicated  factor  of  studying  teams  using  a low- 
fidelity  simulator  is  that  the  generalizability  to  real- 
world  environments  is  limited.  We  tried  to  reconcile  this 
by  developing  an  experimental  team  task  that  contains 
the  activities,  processes  and  situations  that  are  typical  for 
Command  & Control  teams.  Based  on  a generic 
Command  & Control  model  we  performed  a task 
analysis  using  the  previously  described  modelling 
method  that  provides  not  only  a task  hierarchy  but  also 
describes  the  information  dependency  between  tasks,  the 
knowledge  needed  to  perform  tasks  accurately,  and  the 
sequence  of  tasks  for  each  team  member.  Based  on  this 


4-6 


task  analysis  we  specified  the  different  roles  and 
expertise  of  team  members  and  the  information 
dependency  between  them.  In  addition,  by  showing  that 
the  specified  tasks  have  to  be  performed  in  parallel,  we 
demonstrated  that  the  experimental  task  is  a task  for  two 
team  members,  which  cannot  be  performed  individually. 
Based  on  the  task  analysis,  we  attempted  to  develop  an 
experimental  team  task  that  contains  the  advantages  of 
low  fidelity  simulations  but  still  can  be  generalised  to 
real-world  environments. 

The  experimental  task  we  developed  is  a low-fidelity 
simulation  of  a dispatch  centre  representing  a fire- 
fighting organisation  in  a city  (Rasker,  Post  & 
Schraagen,  in  press).  The  fire-fighting  team,  consisting 
of  an  observer  and  an  allocator,  is  required  to  fight  fires 
in  order  to  keep  the  number  of  casualties  as  low  as 
possible,  which  is  the  goal  of  the  task.  In  order  to 
accomplish  the  goal,  the  observer  has  to  assess  the  city 
and  inform  the  allocator  about  the  status  of  the  buildings. 
This  is  comparable  to  the  situation  assessment  function 
in  Command  & Control.  The  allocator  has  to  allocate  a 
number  of  resources  (i.e.,  fire-fighting  units)  to  the 
buildings  to  extinguish  fires.  This  is  comparable  to  the 
function  of  decision-making  and  taking  counter 
measures  in  Command  & Control.  Because  the  number 
of  units  is  limited,  the  team  must  prioritise  and  decide 
upon  which  fires  they  want  to  extinguish.  Team 
members  are  interdependent  of  each  other.  This  means 
that  they  are  required  to  interact  continuously  about  the 
status  of  fires  and  the  resources  to  accomplish  the  goal. 

6 DESIGN 

One  part  of  the  development  of  a new  human-human- 
machine  system,  such  as  a new  platform,  is  the  design  of 
certain  team  organisations  such  as  for  Command  & 
Control,  navigation,  and  ship  control.  Team  design  can 
be  at  a conceptual  level  (eg.,  a proposed  task 
organisation)  or  a can  be  a realisation  at  a detailed  level 
(e  g.,  the  layout  of  a command  centre).  A team 
organisation  may  be  redesigned,  when  bottlenecks  or 
challenges  are  observed  in  an  existing  team.  In  a design 
process,  the  known  team  factors  are  applied  to  new  team 
concepts  and  actual  working  teams. 

In  designing  a human-human  system  the  following 
phases  can  be  differentiated  (Beevis,  Essens  and 
Schuffel,  1995): 

• Defining  the  mission  of  the  overall  system 

• Establishing  the  functions  that  need  to  be  fulfilled 

• Determining  the  tasks  that  need  to  be  carried  out 

• Determining  the  required  manning  and  means 

• Job  design  (task  allocation) 

• Workplace  design 

Project 

The  design  process  is  illustrated  by  the  development  of 
the  command  centre  of  the  Air  Defence  and  Command 


Frigate  of  the  RNLN.  The  first  phases  of  the  design 
process  were  already  established  by  the  RNLN:  the 
mission  to  establish  the  functions  to  fulfil  the  number  of 
agents  needed,  the  choice  of  equipment,  and  job  design. 
Our  role  in  the  design  team  was  to  design  the  workplace 
(Post  & Punte,  1997;  Post  & Punte,  1998). 

From  the  literature  and  our  own  research,  a number  of 
factors  for  team  effectiveness  were  identified: 

• shared  mental  model  (Cannon-Bowers,  Salas  & 
Converse,  1993) 

• non-verbal  communication  (Fussel  & Benimoff,  1995) 

• adaptive  co-ordination  (Entin  & Serfaty,  1999) 

• situation  awareness  (Endsley,  1995) 

• intra  team  feedback  (Rasker  et  al.,  in  press) 

• team  training  (Salas  et  al.,  1993) 

From  these  factors,  and  the  design  decisions  made  by  the 
RNLN  in  the  earlier  phases,  two  concepts  on  team 
support  (as  well  as  other  aspects,  such  as  technical  and 
economical  feasibility)  were  generated  and  assessed. 
Both  concepts  had  a number  of  aspects  in  common  that 
already  were  an  improvement  to  the  most  recent 
command  centre  that  was  designed  for  the  RNLN,  such 
as  the  use  of  large  screen  displays  for  deliberation 
support,  and  a briefing  room  for  training  support.  One 
concept  was  a traditional  one,  based  on  the  most  recent 
command  centre  that  was  designed:  consoles  placed  in 
three  rows,  the  middle  one  the  command  row,  the  row  in 
front  the  picture  compilation  and  weapon  employment 
row,  and  the  row  in  the  back  the  support  row  (e.g., 
communication  support,  equipment  support,  network 
support).  In  the  other  concept,  the  traditional  way  was 
left  and  the  rows  were  placed  in  a circle.  We  found  that 
in  a circle,  the  mean  distance  between  team  members 
was  shortest. 

The  second  rounded  concept  proved  to  be  the  best  at  the 
aspect  of  team  support.  For  individual  w'ork  conditions, 
however,  the  rounded  concept  was  unacceptable  due  to 
adverse  ship  motion  effects  (Bittner  & Guignard.  1985). 
Nonetheless,  a number  of  principles  for  this  concept 
were  taken  over.  This  led  to  the  final  conceptual  design, 
with  large  screen  displays  and  an  integrated  briefing 
room,  that  was  optimised  for  frequent  and  critical 
interactions  (i.e.,  putting  those  people  near  each  other 
who  need  to  work  in  closest  co-operation).  Further, 
visual  deliberation  and  supervision  lines  at  larger 
distances  could  be  optimised  by  placing  the  individual 
work  places  nor  in  a row,  neither  in  a circle,  but  in  a kind 
of  wave  form. 

The  next  step  was  to  translate  the  conceptual  design  in  a 
detailed  design.  At  this  stage,  we  made  a full  scale 
wooden  mock  up  of  both  the  individual  work  places  and 
the  complete  command  centre,  and  invited  in  total  75 
future  users  to  assess  the  design  (among  other  aspects) 
on  co-operation,  deliberation  and  supervision  support. 
Based  on  this  finding  the  command  centre  was 
redesigned  and  approved  for  realisation. 


4-7 


Techniques 

In  designing  the  workplace,  we  used  a number  of 
techniques: 

• link  analysis 

• drawings 

• scenarios 

• physical  mock-ups 

In  RNLN  example,  we  only  designed  the  workplace  of 
the  command  centre.  Currently  we  are  carrying  out 
research  for  the  RNLN  future  Command  & Control,  in 
which  we  also  design  reduced  command  centre  teams, 
including  team  size,  necessary  means,  and  job  design. 

7 EVALUATION 

The  ultimate  goal  of  modelling,  observation, 
experimentation  and  design  is  improving  the 
effectiveness  of  team  performance.  Evaluation  stands  for 
determining  and  verifying  this  effectiveness.  Two  types 
of  measures  are  important  for  team  effectiveness: 
process  measures  and  outcome  measures.  Process 
measures  refer  to  speed  and  workload,  and  outcome 
measures  to  quality  and  quantity. 

Project 

Evaluation  is  not  only  possible  at  a final  stage,  when  a 
design  has  been  realised.  In  the  command  centre 
example  provided  above,  we  used  a wooden  mock  up  for 
evaluation  purposes,  and  had  RNLN  personnel  perform  a 
scenario.  We  asked  them  to  fill  in  a questionnaire  on 
team  aspects.  The  questions  were  about  the  positioning 
of  their  co-workers,  how  the  command  team  was  able  to 
deliberate,  how  supervisors  could  view  their  personnel 
from  their  positions,  and  how  they  could  communicate 
non-verbally.  We  let  them  compare  seven  row 
configurations  (such  as  a straight  row,  an  indented  row 
and  a waved  row)  on  the  team  aspects  mentioned  above. 
Moreover,  we  made  this  comparison  for  three  different 
individual  work  places:  one  with  a single  screen,  one 
with  two  20"  screen,  and  one  with  a primary  20"  screen 
and  a secondary  14"  screen,  making  up  21  (7  times  3) 
configurations.  We  could  find  at  statistical  significance 
level  how  type  of  row  and  type  of  individual  work  place 
relate  to  team  aspects.  Figure  4 shows  some  of  the 
configurations  of  rows. 

Another  evaluation  example  is  obtained  from  the 
SmartStaff  project  (see,  for  an  elaborate  description.  Post 
& Hamaker,  this  volume).  The  aim  of  this  project  is  to 
find  the  best  team  support  for  the  Task  Group  Staff.  This 
staff  is  embarked  on  board  of  the  Air  Defence  and 
Command  Frigate,  and  has  its  own  Staff  room.  In 
contrast  to  the  command  centre  and  the  navigation 
bridge  of  this  frigate,  the  RNLN  had  no  clear  vision  on 
how  to  equip  the  staff  room.  Therefore,  we  took  it  as  a 
research  challenge  to  come  up  with  a support  concept  for 
team  planning,  which  is  the  main  task  of  the  Task  Group 


Staff.  The  resulting  concept,  called  Smartstaff,  supports 
the  team  with: 

• both  individual  and  shared  work  spaces 

• flow  and  storage  of  electronic  information 

• a common  focus  of  attention 

• concurrent  idea  generation 
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Figure  4:  Configurations  of  rows.  The  arrows  show  some 
lines  of  views 

SmartStaff  has  been  evaluated  by  using  a staff  room 
mock-up  with  for  each  team  member  an  individual 
workstation  and  two  large  interactive  touch  screen 
displays,  electronic  storage,  retrieval  and  exchange  of 
information,  and  an  electronic  idea  pad:  a tool  for  both 
individual  and  team  generation  and  representation  of 
ideas.  To  guarantee  face-to-face  contact  and  an  unlimited 
view  of  the  large  screen  display,  the  workstations  were 
lowered  and  positioned  in  a semi-circle  around  the  large 
screen  display. 

Our  aim  was  to  carry  out  a conceptual  evaluation.  At  an 
early  stage,  we  wanted  to  know  whether  our  concept  was 
profitable,  without  having  to  implement  the  concept 
fully.  We  were  able  to  reduce  the  software  development 
effort  to  a minimum,  by  using  existing  software  as  much 
as  possible,  and  letting  the  integration  efforts  (often  the 
most  expensive  parts  of  software  development)  over  to 
the  evaluators,  who  had  therefore  to  strike  some  extra 
keys  during  the  evaluation  session. 

In  this  evaluation  session,  all  14  RNLN  Task  Group 
Staff  members,  including  the  Commodore,  served  as 
subjects.  They  were  first  asked  to  fill  in  a questionnaire 
about  their  current  staff  room.  Next,  they  had  a one-hour 
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training  in  using  the  SmartStaff  environment.  Then,  they 
were  provided  with  a realistic  scenario  in  which  they  had 
to  plan  a journey  for  the  Task  Group.  After  that,  they  had 
to  fill  in  again  a questionnaire,  but  now  about 
SmartStaff.  Having  experienced  this  new  task 
environment,  the  evaluation  session  concluded  with  a 
group  discussion.  During  the  session,  an  external 
obseiver,  who  joined  the  Staff  earlier,  assessed  the  team 
as  well.  It  was  found  that  SmartStaff  provides  better 
support  on  the  unambiguity  of  the  shared  picture  of  a 
tactical  situation  and  the  plan  itself,  on  time 
management,  and  on  presenting  and  communications  of 
ideas  and  plans.  The  quality  of  the  plan  was  not 
influenced. 

Techniques 

Summarising,  the  techniques  used  in  both  studies  are: 

• hardware  and  software  mock-ups 

• Questionnaires 

• Scenarios 

• External  assessor 

• Experienced  group  discussion 

8 SUMMARY  & CONLUSIONS 

Effective  teamwork  is  a prerequisite  in  a complex  system 
such  as  the  command  centre.  Investigating  teamwork  and 
determining  what  makes  a team  effective  is  a great 
challenge  for  team  researchers.  The  objective  of  this 
paper  is  to  describe  in  a framework  how  several  research 
methods  can  be  applied  to  investigate  command  centre 
teams.  This  framework  helps  to  place  the  questions  that 
come  from  team  practice  in  the  set  of  methods  and 
techniques  that  are  currently  available.  The  research 
framework  that  is  proposed  comprises  five  methods  that 
cover  all  aspects  of  team  research:  modelling, 

observation,  experimentation,  design  and  evaluation.  For 
each  method,  the  research  framework  was  illustrated 
with  a number  of  projects  and  the  techniques  Table  1 
gives  an  overview  of  the  methods,  matched  with  the 


questions  for  the  specific  projects,  the  used  techniques, 
and  the  results. 

Matching  the  questions  to  the  methods  makes  it  clear,  in 
our  opinion,  which  research  effort  is  possible  to  provide 
usable  answers.  This  serves  as  a guidance  to  make  a 
choice  between  the  methods  available.  Although  the 
methods  are  treated  separately,  they  are  related  with  each 
other.  Most  important  is  that  they  all  contribute  to  our 
knowledge  of  how  teams  perform  and  which  factors 
make  teams  effective.  This  knowledge  is  used  to  modify 
and  refine  the  research  techniques  and  give  direction  in 
our  new  research  efforts.  Moreover,  this  knowledge  is 
used  to  develop  new  concepts  and  realisations  for 
optimising  team  effectiveness.  We  conclude  that  the 
research  framework  provides  a useful  overview  of  the 
methods  one  can  apply  to  investigate  command  centre 
teams.  Each  method  can  be  used  to  find  an  answer  to  a 
part  of  the  puzzle  how  to  optimise  effective  teamwork. 
In  total,  the  research  framework  represents  an  integral 
approach  for  investigating  command  centres  teams  that 
should  lead  to  an  answer  of  the  complete  puzzle. 

For  future  research  concerning  team  effectiveness  in  the 
command  centre,  all  methods  will  be  applied.  Modelling 
will  be  applied  to  describe  the  command  centre  of  the 
new  mine  counter  measure  vessels  of  the  RNLN.  In 
addition,  we  are  planning  to  elaborate  our  modelling 
techniques  in  order  to  simulate  team  behaviour  in 
Command  & Control  situations.  This  way  we  are  able  to 
use  modelling  not  only  for  description  but  also  for 
experimentation.  Different  factors  will  be  modelled  in 
order  to  investigate  their  effects  on  team  effectiveness. 
Furthermore,  we  will  design  a concept  for  improving  the 
briefing  sessions  to  achieve  more  effective 
communication  in  command  centre  teams.  This  design 
will  also  be  evaluated.  With  this  and  other  research 
efforts,  we  attempt  to  develop  a new  concept  for  future 
Command  & Control  that  improves  team  effectiveness  in 
the  command  centre. 


Modelling 

Observation 

Experimentation 

Design 

Evaluation 

Project 

Multipurpose  frigate 

Multipurpose  frigate 

Intra  team  feedback 

Air  Command  & Defence 
frigate 

Air  Command  & Defence  frigate 
SmartStaff 

Question 

What  is  the 

information  flow  in  the 
command  centre? 

How  can  the 
performance  of  the 
command  centre  be 
optimised? 

What  is  the  effect  of 
intra  team  feedback  on 
developing  shared 
mental  models? 

What  is  the  best  layout  of 
the  command  centre  to 
support  effective  human- 
human  interaction? 

Does  the  design  fulfil  its’ 
expected  effectiveness? 

Techniques 

- document  analyses 

- visualisation 

- interviewing 

- standardisation 

Low  fidelity  simulator: 
fire-fighting  task 

-link  analysis 

- drawings 

- scenarios 
-physical  mock-ups 

- hardware  and  software  mock- 
ups 

- questionnaires 

- scenarios 

- external  assessor 

- experienced  group  discussion 

Result 

clear  description  of  the 
command  centre  with 
400  interrelated 
graphical 
representations 

individual  task 
responsibilities 
dominate  teamwork 
responsibilities 

teams  perform  better 
when  they  have  the 
opportunity  to  engage 
in  intra  team  feedback 

complete  workplace  layout 
supporting  team 
effectiveness 

- workplace  layout  mock-up 

- SmartStaff  concept  supporting 
collaborative  decision-making 
and  planning 

Table  1 : overview  of  the  methods,  projects,  questions,  techniques  and  results 
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